Method and wearable device for providing a virtual input interface

ABSTRACT

Provided is a wearable device including: an image sensor configured to sense a gesture image of a user setting a user input region; and a display configured to provide a virtual input interface corresponding to the set user input region.

RELATED APPLICATIONS

This application is a Continuation Application of U.S. application Ser.No. 15/782,505, filed Oct. 12, 2017, which is a Continuation applicationof U.S. application Ser. No. 14/665,678, filed Mar. 23, 2015, whichclaims priority from Korean Patent Application No. 10-2014-0033705,filed on Mar. 21, 2014, Korean Patent Application No. 10-2014-0098653,filed on Jul. 31, 2014, and Korean Patent Application No.10-2014-0179354, filed on Dec. 12, 2014, in the Korean IntellectualProperty Office, the disclosures of which are incorporated herein byreference in their entireties.

BACKGROUND 1. Field

One or more exemplary embodiments relate to a method and wearable devicefor providing a virtual input interface.

2. Description of Related Art

The real world is a space consisting of 3-dimensional (3D) coordinates.People are able to recognize 3D space by combining visual informationobtained using two eyes. However, a photograph or a moving imagecaptured by a general digital device is expressed in 2D coordinates, andthus does not include information about space. In order to give afeeling of space, 3D cameras or display products that capture anddisplay 3D images by using two cameras have been introduced.

Meanwhile, a current input method of smart glasses is limited. A userbasically controls the smart glasses by using a voice command. However,it is difficult for the user to control the smart glasses by using onlya voice command if a text input is required. Thus, a wearable systemthat provides various input interaction methods is required.

SUMMARY

Methods and apparatuses consistent with exemplary embodiments include amethod and wearable device for setting an input region in the air or onan actual object based on a user motion, and providing a virtual inputinterface in the set input region.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented exemplary embodiments.

According to one or more exemplary embodiments, a wearable deviceincludes: an image sensor configured to sense a gesture image of a usersetting a user input region; and a display configured to provide avirtual input interface corresponding to the user input region set byusing the sensed gesture image.

The sensed gesture image may correspond to a figure drawn by the user,and the virtual input interface may be displayed to correspond to thesensed figure.

The virtual input interface may be displayed to correspond to a size ofthe user input region.

The virtual input interface may be determined based on a type of anapplication being executed by the glasses type wearable device.

The display may include a transparent display configured to provide thevirtual input interface on a region of the transparent displaycorresponding to the user input region as observed through thetransparent display.

The image sensor may be configured to capture a first image of the userinput region, and the display may be configured to display a secondimage of the virtual input interface over the user input region of thefirst image.

The glasses type wearable device may further include: a depth sensorconfigured to sense a first depth value corresponding to a distance fromthe wearable device to the user input region, and a second depth valuecorresponding to a distance from the wearable device to an input tool;and a controller configured to determine whether an input is generatedthrough the virtual input interface based on the first depth value andthe second depth value.

The displayed size of the virtual input interface may be determinedbased the first depth value.

The controller may be configured to determine that an input is generatedthrough the virtual input interface when a difference between the firstand second depth values is less than a threshold value.

The controller may be configured to determine that an input is generatedthrough the virtual input interface when the second depth value isgreater than the first depth value.

According to one or more exemplary embodiments, a method of providing,by a wearable device, a virtual input interface, includes: obtaining agesture image of a user for setting a user input region; and providing avirtual input interface corresponding to the user input region such thatthe virtual input interface corresponds to a size of the user inputregion.

The obtaining of the gesture image may include: obtaining the gestureimage by recognizing a figure drawn by the user; and setting a regioncorresponding to the figure as the user input region.

The virtual input interface may be determined based on a size of theuser input region.

The method may further include determining the virtual input interfacebased on a type of object where the user input region is set.

The method may further include determining the virtual input interfacebased on a type of an application being executed by the wearable device.

The virtual input interface may be provided on a transparent displaysuch that the virtual input interface corresponds to the user inputregion as observed through the transparent display.

The providing of the virtual input interface may include: capturing afirst image of the user input region by using an image sensor;generating a second image of the virtual input interface; and displayingthe second image over the user input region of the first image.

The method may further include: obtaining a first depth valuecorresponding to a distance from the wearable device to the user inputregion, and a second depth value corresponding to a distance from thewearable device to an input tool; and determining whether an input isgenerated through the virtual input interface based on the first depthvalue and the second depth value.

A displayed size of the virtual input interface may be determined basedon a size of the user input region.

The determining of whether the input is generated may includedetermining that a difference between the first and second depth valuesis less than a threshold value.

The determining of whether the input is generated may includedetermining the second depth value is greater than the first depthvalue.

According to one or more exemplary embodiments, a wearable input deviceincludes: a sensor configured to sense a plurality of gestures and areal world image; a display configured to display a graphic userinterface; and a controller configured to determine an input region ofthe real world image, control the display to display the graphic userinterface on an area corresponding to the determined input region, anddetermine an input based on an input gesture of the plurality ofgestures.

The wearable input device may include a communicator configured toreceive a touch signal from an external device. The controller may befurther configured to determine the input based on the touch signal.

The may be further determined based on an input region defining gestureof the plurality of gestures.

The sensor may be further configured to determine a distance between thewearable input device and the input region.

The controller may be further configured to continuously update adisplay region of the graphic user interface based on the real worldimage.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects will become apparent and more readilyappreciated from the following description of the exemplary embodiments,taken in conjunction with the accompanying drawings in which:

FIGS. 1A through 1E are diagrams describing a system for providing, by awearable device, a virtual input interface, according to an exemplaryembodiment;

FIG. 2 is a flowchart illustrating a method of providing, by a wearabledevice, a virtual input interface, according to an exemplary embodiment;

FIGS. 3A through 5B are diagrams describing methods of setting an inputregion, according to exemplary embodiments;

FIG. 6 is a flowchart illustrating a method of providing a virtual inputinterface according to a depth value of an input region, according to anexemplary embodiment;

FIGS. 7 through 9 are diagrams describing a type and a size of a virtualinput interface being changed according to a depth value of an inputregion, according to an exemplary embodiment;

FIGS. 10A and 10B are diagrams describing a type of a virtual inputinterface being adaptively changed according to a change of a depthvalue of an actual object where an input region is set, according toexemplary embodiments;

FIGS. 10C and 10D are diagrams describing a type of a virtual inputinterface being changed based on a user input, according to exemplaryembodiments;

FIG. 11 is a flowchart illustrating a method of providing a virtualinput interface determined based on a size of an input region or asetting motion of the input region, according to an exemplaryembodiment;

FIGS. 12A through 13B are diagrams describing types of virtual inputinterfaces being changed according to dimensions of input regions;

FIGS. 14A through 15B are diagrams describing types of virtual inputinterfaces being changed according to gestures setting input regions;

FIGS. 16A and 16B are diagrams describing providing a virtual inputinterface determined based on an object where an input region is set,according to an exemplary embodiment;

FIGS. 17A through 17C are diagrams describing a virtual input interfaceprovided by a wearable device, the virtual input interface determinedbased on a type of an actual object where an input region is set,according to an exemplary embodiment;

FIGS. 18A and 18B are diagrams describing a virtual input interface, thevirtual input interface determined based on an input tool that sets aninput region, according to an exemplary embodiment;

FIG. 19 is a flowchart illustrating a method of providing a virtualinput interface determined based on an application being executed by awearable device, according to an exemplary embodiment;

FIGS. 20A and 20B are diagrams describing providing a virtual inputinterface determined based on a type of an application being executed,according to an exemplary embodiment;

FIG. 21 is a diagram describing a virtual input interface determinedbased on a type of content being executed, according to an exemplaryembodiment;

FIGS. 22A through 23B are diagrams describing virtual input interfacesthat are the same as previous virtual input interfaces provided when awearable device recognizes actual objects where the previous virtualinput interfaces were provided, according to exemplary embodiments;

FIG. 24 is a flowchart illustrating a method of providing a virtualinput interface in an input region set in the air, according to anexemplary embodiment;

FIGS. 25A and 25B are diagrams describing a method of determiningwhether an input is generated through a virtual input interface, when aninput region is set in the air;

FIG. 26 is a flowchart illustrating a method of providing a virtualinput interface in an input region set in the air or on an actualobject, according to an exemplary embodiment;

FIGS. 27A and 27B are diagrams describing a method of determiningwhether an input is generated through a virtual input interface, when aninput region is set on an actual object;

FIGS. 28A and 28B are diagrams describing a method of obtaining a firstdepth value of an input region and a second depth value of an inputtool, according to an exemplary embodiment;

FIG. 29 is a flowchart illustrating a method of providing feedback aboutwhether an input is generated through a virtual input interface,according to an exemplary embodiment;

FIGS. 30 and 31 are diagrams describing outputting notification signalscorresponding to whether inputs are generated by wearable devices,according to exemplary embodiments;

FIG. 32 is a diagram describing outputting of a notification signalcorresponding to whether an input is generated through a virtual inputinterface, according to an exemplary embodiment; and

FIGS. 33 and 34 are block diagrams of a wearable device according toexemplary embodiments.

DETAILED DESCRIPTION

Terms used in the present specification will be briefly described andone or more exemplary embodiments will be described in detail.

All terms including descriptive or technical terms which are used hereinshould be construed as having meanings that are obvious to one ofordinary skill in the art. However, the terms may have differentmeanings according to an intention of one of ordinary skill in the art,precedent cases, or the appearance of new technologies. Also, some termsmay be arbitrarily selected by the applicant, and in this case, themeaning of the selected terms will be described in detail in thedetailed description of the invention. Thus, the terms used herein haveto be defined based on the meaning of the terms together with thedescription throughout the specification.

Also, when a part “includes” or “comprises” an element, unless there isa particular description contrary thereto, the part can further includeother elements, not excluding the other elements. In the followingdescription, terms such as “unit” and “module” indicate a unit forprocessing at least one function or operation, wherein the unit and theblock may be embodied as hardware or software or embodied by combininghardware and software.

As used herein, the term “and/or” includes any and all combinations ofone or more of the associated listed items. Expressions such as “atleast one of,” when preceding a list of elements, modify the entire listof elements and do not modify the individual elements of the list.

One or more exemplary embodiments will now be described more fully withreference to the accompanying drawings. However, the one or moreexemplary embodiments may be embodied in many different forms, andshould not be construed as being limited to the exemplary embodimentsset forth herein; rather, these exemplary embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the concept of the one or more exemplary embodiments to those ofordinary skill in the art. In the following description, well-knownfunctions or constructions are not described in detail because theywould obscure the one or more exemplary embodiments with unnecessarydetail. Like reference numerals in the drawings denote like or similarelements throughout the specification.

FIGS. 1A through 1E are diagrams describing a system for providing, by awearable device 100, a virtual input interface, according to anexemplary embodiment.

The wearable device 100, according to an exemplary embodiment, mayinclude a head mounted display (HMD) that is mountable on a headportion. For example, the HMD may be glasses, a helmet, or a hat, but isnot limited thereto. The first wearable device 100, according to anexemplary embodiment, may be a watch, a band, a ring, a necklace, abracelet, a shoe, an earring, a headband, clothes, a glove, or athimble.

The wearable device 100, according to an exemplary embodiment, may beone device or a combination of a plurality of devices. For example, thewearable device 100 may be glasses, or a combination of at least twodevices, such as glasses and a ring, glasses and a watch, or glasses anda thimble.

The wearable device 100, according to an exemplary embodiment, mayprovide at least one virtual input interface. For example, the wearabledevice 100, according to an exemplary embodiment, may display a virtualinput interface on an optical display 121, such that the virtual inputinterface matches the real world observed through the optical display121.

A structure of the optical display 121 will now be described in detailwith reference to FIG. 1B.

Referring to FIG. 1B, the optical display 121 may include a displaydevice 210 and a light guide 200 a. The light guide 200 a may include alight guiding device 220 a and a variable lens 240 a. Also, the displaydevice 210 may output a first light 201 forming an image to the lightguiding device 220 a. The display device 210 may have a quadrangularplate shape, and may display an image in a pixel unit according to datainput from a controller. For example, the display device 210 may be alight-emitting diode (LED), an organic LED (OLED), a liquid crystaldisplay (LCD), or a liquid crystal on silicon (LCOS).

The light guiding device 220 a may include first through fifth surfaces221 through 225 a. The light guiding device 220 a may guide the firstlight 201 input from the display device 210 towards the variable lens240 a via internal reflection or total internal reflection.

The first surface 221 corresponds to a part of a rear surface of thelight guiding device 220 a, which faces the display device 210, and maytransmit the first light 201 input from the display device 210 towardsthe second surface 222. The second surface 222 corresponds to a firstside surface of the light guiding device 220 a between the first andthird surfaces 221 and 223, and may reflect the first light 201penetrated through the first surface 221 towards the third or fourthsurface 223 or 224.

The third surface 223 corresponds to a front surface of the lightguiding device 220 a, the fourth surface 224 corresponds to a remainingpart of the rear surface of the light guiding device 220 a, and thethird and fourth surfaces 223 and 224 each reflect or totally reflectthe first light 201 such that the first light 201 reaches the fifthsurface 225 a. Here, total reflection means that the first light 201incident on an interface (i.e., the third or fourth surface 223 or 224)of the light guiding device 220 a and an external air layer from aninside of the light guiding device 220 a is totally reflected withoutpenetration at the interface.

The fifth surface 225 a corresponds to a second side surface of thelight guiding device 220 a between the third and fourth surfaces 223 and224, and may transmit the first light 201 towards the variable lens 240a and reflect the first light 201 incident from the variable lens 240 atowards eyes of a user. The fifth surface 225 a may transmit a secondlight 202 forming a front view of the first wearable device 100 towardsthe eyes of the user.

The light guiding device 220 a may include a body portion 232 a that isdisposed between the third and fourth surfaces 223 and 224 and has auniform thickness, a first slope portion 231 that is disposed betweenthe first and second surfaces 221 and 222 and has a thickness graduallydecreasing away from the body portion 232 a, and a second slope portion233 a that is disposed between the third and fourth surfaces 223 and 224and has a thickness gradually decreasing away from the body portion 232a. The second slope portion 233 a may have the fifth surface 225 a thatis an inclined surface facing the variable lens 240 a and the eyes ofthe user.

The variable lens 240 a may include a penetration surface 241 throughwhich the first light 201 penetrates, a refraction surface 242 thatrefracts the first light 201, and the reflection surface 243 a thatreflects the first light 201. A shape or curvature of the refractionsurface 242 may change according to control of the controller. Thevariable lens 240 a may adjust a virtual object distance from the eyesof the user to a virtual object by adjusting an angle (i.e., an incidentangle) of the first light 201 incident on the eyes of the user accordingto a change of the shape or curvature of the refraction surface 242.

FIGS. 1C and 1D are diagrams describing adjusting a distance of avirtual input interface by using the variable lens 240 a, according toan exemplary embodiment.

The variable lens 240 a may adjust a distance from an eye 30 of a userto a virtual input interface 41 recognized by the user, by adjusting anincident angle of a first light 43 incident on the eye 30 according tocontrol of the controller.

Referring to FIG. 1C, a thickness of an eye lens 31 decreases to focusthe eye 30 on an actual object 34 at a long distance. A second light 35starting from an actual object 34 moves in parallel to an optical axis33 of the eye 30, is incident on the eye lens 31 through the fifthsurface 225 a of the light guiding device 220 a, and is converged on aretina 32 by being refracted at the eye lens 31. In other words, the eyelens 31 forms an image of the actual object 34 on the retina 32.

The variable lens 240 a may transmit the first light 43 to the fifthsurface 225 a. The first light 43 reflected at the fifth surface 225 amoves in parallel to the optical axis 33 of the eye 30 to be incident onthe eye lens 31, and the eye lens 31 may refract the first light 43 tobe converged on the retina 32. In other words, the eye lens 31 may forman image of the virtual input interface 41 on the retina 32. Forexample, when the actual object 34 (or the image of the actual object34) is in an in-focus state, the actual object 34 (or the image of theactual object 34) and the virtual input interface 41 (or the image ofthe virtual input interface 41) may have the same first object distanceOD1 and the same image distance ID.

Referring to FIG. 1D, the thickness of the eye lens 31 increases tofocus the eye 30 on an actual object 36 at a short distance. A secondlight 37 starting from the actual object 36 moves along the optical axis33 of the eye 30 while diverging (or diffusing), is incident on the eyelens 31 through the fifth surface 225 a of the light guiding device 220a, and is converged on the retina 32 by being refracted by the eye lens31. In other words, the eye lens 31 forms an image of the actual object36 on the retina 32. The variable lens 240 a may transmit a first light44 to the fifth surface 225 a. The first light 44 reflected from thefifth surface 225 a is incident on the eye lens 31 by moving along theoptical axis 33 of the eye 30 while diverging (or diffusing), and theeye lens 31 may refract the first light 44 to be converged on the retina32. In other words, the eye lens 31 may form an image of a virtual inputinterface 42 on the retina 32. For example, when the actual object 36(or the image of the actual object 36) is in an in-focus state, theactual object 36 (or the image of the actual object 36) and the virtualinput interface 42 (or the image of the virtual input interface 42) mayhave the same second object distance OD2 and the image distance ID.

Meanwhile, the wearable device 100, according to an exemplaryembodiment, may recognize a motion of an input tool for setting an inputregion, and provide a virtual input interface determined based onattributes of the input region, as will be described in detail laterwith reference to FIG. 2.

Referring to FIG. 1E, a virtual input interface 50, according to anexemplary embodiment, may be a graphical user interface (GUI) forreceiving an input of a user using the first wearable device 100.Alternatively, the virtual input interface 50 may be realized in any oneof various forms, and for example, the virtual input interface 50 may bea keyboard (such as a QWERTY keyboard or a portable terminal keyboard),a memo pad, a game controller, a calculator, a plano keyboard, a drum,or a dial pad, but is not limited thereto.

The wearable device 100, according to an exemplary embodiment, mayprovide the virtual input interface 50 on the input region set by theuser. The wearable device 100 may display the virtual input interface 50on the optical display 121 such that the virtual input interface 50overlaps the input region.

Here, the wearable device 100 may display the virtual input interface 50on the optical display 121 in the form of an augmented reality (AR), amixed reality (MR), or a virtual reality (VR).

For example, when the virtual input interface 50 is provided in the formof AR or MR, the wearable device 100 may display the virtual inputinterface 50 on a transparent display such that the virtual inputinterface 50 overlaps the input region observed through the transparentdisplay.

As shown in FIG. 1E, a region 20 defined by a dashed line denotes aregion of the real world observed through the optical display 121 of thewearable device 100. The wearable device 100 may display the virtualinput interface 50 on the optical display 121 such that the virtualinput interface 50 matches the region 20 observed through the opticaldisplay 121.

Alternatively, when the virtual input interface 50 is provided in theform of VR, the wearable device 100 may capture a first image includingan input region set in the real world, and generate a second image byadding the virtual input interface 50 to the input region of the firstimage. The wearable device 100 may display the second image in which thevirtual input interface 50 overlaps the input region on an opaquedisplay.

The wearable device 100, according to an exemplary embodiment, mayinclude an image sensor 111 and a depth sensor 112.

The image sensor 111 may capture an external image or detect a usermotion setting an input region. Also, the image sensor 111 may detectmovement of an input tool. Here, the input toll may be a pre-set tool,and examples of the input tool include a pen, a finger, a stylus and astick, but are not limited thereto.

The depth sensor 112 may measure a depth value of the input region setby the user or a depth value of the input tool. A depth value maycorrespond to a distance from the depth sensor 112 to a certain object.In the present specification, the depth value increases as the distancefrom the depth sensor 112 to the certain object increases.

For example, the depth value may be the distance from the depth sensor112 to the certain object on a Z-axis. As shown in FIG. 1A, in a 3Dspace, an X-axis may be a reference axis passing the wearable device 100from left to right, an Y-axis may be a reference axis passing thewearable device 100 from top to bottom, and the Z-axis may be areference axis passing the wearable device 100 from back to front. Also,the X-, Y-, and Z-axes may be perpendicular to each other.

According to an exemplary embodiment, the depth sensor 112 may obtain adepth value of an object via any one of various methods. For example,the depth sensor 112 may measure a depth value by using at least one ofa time of flight (TOF) method, a stereoscopic vision method, and astructured light pattern method.

The TOF method is a method of measuring a distance to an object byanalyzing a time consumed before light returns after being reflected atthe object. In a TOF system, an infrared LED irradiates an infraredlight pulse, and an infrared camera measures a time before the infraredlight pulse returns after being reflected at an object. In this case,the depth sensor 112 may include the infrared LED and the infraredcamera. The depth sensor 112 may repeatedly irradiate and receive lightdozens of times per second to obtain distance information in the form ofa moving image. Also, the depth sensor 112 may generate a depth mapindicating distance information representing brightness of color of eachpixel.

The stereoscopic vision method is a method of obtaining a 3D effect ofan object by using two cameras. Accordingly, the depth sensor 112 mayinclude two cameras. The depth sensor 112 may calculate a distance basedon triangulation, by using difference information of images captured bythe two cameras. A person feels a 3D effect through a difference betweenimages viewed by left and right eyes, and the depth sensor 112 measuresa distance in the same manner as eyes of a person. For example, when adistance is short, a difference between images captured by two camerasis high, and when a distance is long, a difference between imagescaptured by two cameras is low.

The structured light pattern method is a method of illuminating anobject with a patterned light and measuring a distance to the object byanalyzing a location of a pattern on a surface of the object. The depthsensor 112 generally projects a linear pattern or a dot pattern on anobject, and the linear pattern or the dot pattern varies based on curvesof the object.

The structured light pattern method may be performed by replacing one ofthe two cameras used in the stereoscopic vision method with a lightprojector. For example, the depth sensor 112 may calculate a depth mapin real-time by analyzing an algorithm of locations of patternsgenerated as a light emitted from an infrared projector incident on asurface of an object.

Meanwhile, the image sensor 111 and the depth sensor 112 may be separatesensors, or configured as one sensor.

The wearable device 100, according to an exemplary embodiment, maydetermine whether an input is generated through the virtual inputinterface 50 by using a depth value of an input region or an input toolobtained through the image sensor.

FIG. 2 is a flowchart of a method of providing, by the wearable device100, a virtual input interface, according to an exemplary embodiment.

Referring to FIG. 2, the wearable device 100 may set an input region inoperation S210. The input region may be a 2D or 3D space of the realworld on which a virtual input interface overlaps when the virtual inputinterface is displayed on the optical display 121.

The wearable device 100 may set the input region based on a user motion.For example, the wearable device 100 may recognize a figure drawn by auser in the air or on an actual object, such as a palm, a desk, or awall, by using an input tool, such as a finger, a pen, a stylus or astick, and set a region corresponding to the figure as the input region.

Alternatively, the wearable device 100 may recognize a pre-set objectand set a region corresponding to the pre-set object as the inputregion. Alternatively, the wearable device 100 may recognize a movementof the user touching a pre-set object by using the input tool, and set aregion corresponding to the pre-set object as the input region.

A method of setting an input region will be described in detail laterwith reference to FIGS. 3A through 5B.

Also, the wearable device 100, according to an exemplary embodiment, mayreceive a pre-set voice input or a pre-set key input for entering aninput region setting mode. For example, when a voice input or a keyinput for entering an input mode is received, the wearable device 100may be controlled to obtain a user gesture image for setting the inputregion. Alternatively, when an application that requires an input isexecuted, the wearable device 100 may be controlled to obtain the usergesture image for setting the input region.

When the input region is set, the wearable device 100 may determine avirtual input interface to be displayed based on attributes of the inputregion, in operation S220.

For example, the wearable device 100 may determine the virtual inputinterface to be displayed on the optical display 121 based on at leastone of a size of the input region, a shape of the input region, adistance between the input region and the wearable device 100 (a depthvalue of the input region), a type of an actual object where the inputregion is set, and a gesture of setting the input region.

The wearable device 100 may display the virtual input interface tooverlap the input region in operation S230.

Here, the wearable device 100 may display the virtual input interface inthe form of AR, MR, or VR.

For example, when the virtual input interface is displayed in the formof AR or MR, the wearable device 100 may display the virtual inputinterface on a transparent display, such as a see-through type display,such that the virtual input interface overlaps the input region (the 2Dor 3D space of the real world) observed through the transparent display.

Alternatively, when the virtual input interface is displayed in the formof VR, the wearable device 100 may capture a first image (an actualimage) including the input region (the 2D or 3D space of the realworld), and generate a second image by adding the virtual inputinterface (a virtual image) to the input region of the first image. Thewearable device 100 may display the second image in which the virtualinput interface overlaps the input region on an opaque display, such asa see-close type display.

The wearable device 100, according to an exemplary embodiment, mayobtain a first depth value of the input region and a second depth valueof the input tool touching the virtual input interface, in operationS240.

The wearable device 100 may measure a distance (the depth value of theinput region, i.e., the first depth value) from the wearable device 100to the input region by using the depth sensor 112.

Meanwhile, when the input region does not exist on the same plane, aplurality of depth values of the input region may exist. When theplurality of depth values of the input region exist, the first depthvalue may be one of an average depth value of the plurality of depthvalues, a minimum depth value of the plurality of depth values, and amaximum depth value of the plurality of depth values, but is not limitedthereto.

When the input region is set on an actual object, the first depth valuemay be a depth value of the actual object.

The wearable device 100 may measure a distance (the depth value of theinput tool, i.e., the second depth value) from the wearable device 100to the input tool by using the depth sensor 112.

When the input tool is a 3D object, a plurality of depth values of theinput tool may exist. When the plurality of depth values of the inputtool exist, the second depth value may be one of an average depth valueof the plurality of depth values, a minimum depth value of the pluralityof depth values, and a maximum depth value of the plurality of depthvalues, but is not limited thereto.

For example, when the virtual input interface is touched by the inputtool, a point where the input tool and the virtual input interfacecontact each other (an end point of the input tool) may be the seconddepth value.

The wearable device 100 may determine whether an input is generatedthrough the virtual input interface by comparing the first and seconddepth values in operation S250.

For example, the first depth value of the input region may be areference value for determining whether an input is generated, and thewearable device 100 may determine that the input is generated throughthe virtual input interface when a difference between the first andsecond depth values is less than a threshold value.

Alternatively, the wearable device 100 may determine that the input isgenerated through the virtual input interface when the second depthvalue is greater than the first depth value.

The wearable device 100, according to an exemplary embodiment, may setan input region based on a user motion, and determine whether an inputis generated by comparing a depth value of the input region and a depthvalue of an input tool to increase accuracy of the input through avirtual input interface.

FIGS. 3A through 5B are diagrams describing methods of setting an inputregion, according to exemplary embodiments.

Referring to FIGS. 3A and B, the wearable device 100, according to anexemplary embodiment, may set an input region by recognizing a figuredrawn by a user in the air or on an actual object.

For example, as shown in FIG. 3A, the user may draw a figure, such as arectangle, in the air by using an input tool 310, such as a pen, astick, a stylus or a finger. The wearable device 100 may recognize thefigure and set a region corresponding to the figure as an input region320. For example, a region having a depth value of the figure (adistance from the wearable device 100 to the figure), a shape of thefigure, and a size of the figure may be set as the input region 320.

As shown in FIG. 3A, the figure may be a rectangle, but a shape of thefigure is not limited thereto. Examples of the figure include figureshaving various shapes and sizes, such as a circle, a polygon, and a freelooped curve, a 2D figure, and a 3D figure.

Alternatively, as shown in FIG. 3B, the user may draw a FIG. 340, suchas a rectangle, on an actual object 330, such as a palm, by using aninput tool 345, such as a pen, a stick, a stylus or a finger. Thewearable device 100 may recognize the FIG. 340 drawn by the user and seta region corresponding to the FIG. 340 as an input region. For example,a region having a depth value of the FIG. 340 (a distance from thewearable device 100 to the actual object 330), a shape of the FIG. 340,and a size of the FIG. 340 may be set as the input region.

Referring to FIGS. 4A and 4B, the wearable device 100, according to anexemplary embodiment, may set an input region by recognizing a certainobject.

For example, as shown in FIG. 4A, the wearable device 100 may recognizea palm 410 by using the image sensor 111. Here, information about ashape or size of the palm 410 may be pre-stored in the wearable device100. Accordingly, the wearable device 100 may compare the shape and sizeof the palm 410 with the pre-stored information, and determine whetherto set the palm 410 as the input region.

When the shape and size of the palm 410 are the same as the pre-storedinformation, the wearable device 100 may set a pre-set region 420 of thepalm 410 as an input region. Here, a shape and size of the pre-setregion 420 may vary.

As shown in FIG. 4A, the wearable device 100 may recognize the palm 410and sets the input region. Alternatively, the wearable device 100 mayset the input region by recognizing any one of various objects, such asa desk and a notepad.

Also, the wearable device 100 may define a certain shape as a marker,and set a plane of an actual object including the marker as an inputregion when the marker is recognized.

For example, when a rectangle is defined as the marker, the wearabledevice 100 may recognize the rectangle as the marker by using the imagesensor 111. As shown in FIG. 4B, the wearable device 100 may recognize anotepad in a rectangle 430 as the marker.

When the marker is recognized, the wearable device 100 may set a planeof an actual object including the marker as an input region. Forexample, as shown in FIG. 4B, the wearable device 100 may set a plane ofthe notepad in the rectangle 430 as an input region. Here, the wearabledevice 100 may set an entire plane of the notepad as the input region,or a partial region of the plane of the notepad as the input region.

As shown in FIG. 4B, a rectangle may be defined as the marker.Alternatively, any one of various shapes, such as a circle and apolygon, may be defined as the marker.

Referring to FIG. 5A, the wearable device 100, according to an exemplaryembodiment, may set an input region by recognizing an actual inputinterface.

The wearable device 100 may recognize the actual input interface anddisplay a virtual input interface having the same type as the actualinput interface. Also, the wearable device 100 may receive an input oftouching the actual input interface by a user using an input tool 520,such as a pen, a stick, a stylus or a finger, and then recognize theactual input interface.

Examples of the actual input interface include an actual keyboard, anactual keypad, an actual notepad interface, an actual calculator, anactual plano keyboard, an actual game controller, and an actual dialpanel, but are not limited thereto. Alternatively, the actual inputinterface may be a GUI displayed on a mobile terminal.

For example, as shown in FIG. 5A, when the user touches an actualkeyboard 510 by using the input tool 520, the wearable device 100 mayrecognize the actual keyboard 510 touched by the input tool 520. At thistime, the wearable device 100 may obtain a depth value of the actualkeyboard 510 and a depth value of the input tool 520 by using the depthsensor 112, and determine that the actual keyboard 510 is touched when adifference between the depth value of the actual keyboard 510 and thedepth value of the input tool 520 is equal to or less than a thresholdvalue.

Also, information about a type, shape, and size of one or more actualinput interfaces may be pre-stored in the wearable device 100.Accordingly, the wearable device 100 may compare a type, shape, and sizeof the actual keyboard 510 recognized by the image sensor 111 with thepre-stored information, and determine whether the actual keyboard 510 isan actual input interface.

Also, the wearable device 100 may display a virtual input interface thatcorresponds to the actual input interface. The wearable device 100 maydisplay the virtual input interface having the same size and shape asthe actual input interface on the optical display 121 such that thevirtual input interface overlaps a region of the actual input interface.

For example, as shown in FIG. 5A, when the actual keyboard 510 isrecognized, the wearable device 100 may display a virtual keyboardhaving the same size and shape as the actual keyboard 510 such that thevirtual keyboard overlaps a region where the actual keyboard 510 isdisplayed.

Meanwhile, referring to FIG. 5B, the wearable device 100, according toan exemplary embodiment, may recognize a plane of an actual object andset an input region.

The wearable device 100 may recognize the plane of the actual object,and when the user touches the plane by using an input tool, such as apen, a stick, a stylus or a finger, the wearable device 100 may set thetouched plane as the input region.

For example, as shown in FIG. 5B, when the user touches a plane 540 of anotepad by using an input tool 530, such as a pen, the wearable device100 may recognize the plane 540 of the notepad touched by the input tool530. Here, the wearable device 100 may obtain a depth value of the plane540 and a depth value of the input tool 530 by using the depth sensor112, and determine that the input tool 530 touched the plane 540 when adifference between the depth value of the plane 540 and the depth valueof the input tool 530 is equal to or less than a threshold value.

Accordingly, the wearable device 100 may set the plane 540 touched bythe input tool 530 as an input region.

FIG. 6 is a flowchart illustrating a method of providing a virtual inputinterface according to a depth value of an input region, according to anexemplary embodiment.

Referring to FIG. 6, the wearable device 100 may set an input regionbased on a user motion, in operation S610. Because operation S610 hasbeen described above in detail with reference to operation S210 of FIG.2, and FIGS. 3A through 5B, details thereof are not repeated.

The wearable device 100 may obtain a first depth value of the inputregion in operation S620.

When the input region is set in the air, the wearable device 100 mayobtain a depth value of the input region based on the user motion ofsetting the input region. For example, when a user draws a figure in theair by using an input tool, the wearable device 100 may obtain a depthvalue of the input tool drawing the figure by using the depth sensor112, and set the depth value of the input tool as the first depth valueof the input region.

Alternatively, when the input region is set on an actual object, thewearable device 100 may obtain a depth value of the actual object byusing the depth sensor 112, and set the depth value of the actual objectas the first depth value of the input region.

The wearable device 100 may determine a type of a virtual inputinterface to be displayed, based on the first depth value of the inputregion, in operation S630.

For example, when the first depth value of the input region is equal toor less than a first threshold value, the wearable device 100 maydetermine a first keyboard having a first size as the virtual inputinterface to be displayed on the optical display 121.

Also, when the first depth value of the input region is greater than thefirst threshold value and equal to or less than a second threshold valuethat is greater than the first threshold value, the wearable device 100may determine a second keyboard having a second size as the virtualinput interface to be displayed on the optical display 121, wherein thesecond size is smaller than the first size.

Also, when the first depth value of the input region is greater than thesecond threshold value, the wearable device 100 may determine a thirdkeyboard having a third size as the virtual input interface to bedisplayed on the optical display 121, wherein the third size is smallerthan the second size.

When the first depth value of the input region increases, a size of theinput region observed by the user of the wearable device 100 decreases,and thus the wearable device 100 may determine a virtual input interfacehaving a relatively smaller size. However, an exemplary embodiment isnot limited thereto.

Also, the wearable device 100 may not only determine the size of thevirtual input interface, but also a shape of the virtual input interfacebased on the first depth value of the input region, as will be describedin detail later with reference to FIGS. 7 through 9.

Referring back to FIG. 6, in operation S640, the wearable device 100 maydisplay the virtual input interface determined in operation S630 on theoptical display 121 such that the virtual input interface overlaps theinput region set in operation S610.

Also, the wearable device 100 may obtain a second depth value of theinput tool touching the virtual input interface in operation S650, andcompare the first and second depth values to determine whether an inputis generated through the virtual input interface in operation S660.

Because operations S640 through S660 of FIG. 6 have been described indetail above with reference to FIGS. S230 through S250, details thereofare not repeated.

FIGS. 7 through 9 are diagrams describing a type and a size of a virtualinput interface displayed on the optical display 121 being changedaccording to a depth value of an input region.

Referring to FIG. 7, the wearable device 100 may recognize a gesture(for example, a gesture of drawing a rectangle) of a user setting aninput region on a palm 710 that is 7 cm away from the wearable device100, by using an input tool, such as a finger, a pen, a stylus or astick. The wearable device 100 may display a QWERTY keyboard 720 on theoptical display 121 such that the QWERTY keyboard 720 matches the palm710 observed through the optical display 121, based on the gesture.Here, as shown in FIG. 7, the QWERTY keyboard 720 may include an inputwindow (a window displaying ‘input message’), and text input through theQWERTY keyboard 720 may be displayed on the input window.

Also, referring to FIG. 8, the wearable device 100 may recognize agesture (for example, a gesture of drawing a rectangle) of a usersetting an input region on a palm 810 that is 10 cm away from thewearable device 100, by using an input tool, such as a finger, a pen, astylus or a stick.

When a distance between the palm 810 and the wearable device 100 is 10cm, a size of the palm 810 observed through the optical display 121 maybe smaller than that of the palm 710 of FIG. 7 that is 7 cm away fromthe wearable device 100. Accordingly, the wearable device 100 maydisplay a mobile terminal keyboard 820, such as Cheonjiin keyboard, onthe optical display 121 such that the mobile terminal keyboard 820matches the palm 810 observed through the optical display 121.

Also, referring to FIG. 9, the wearable device 100 may recognize agesture (for example, a gesture of drawing a rectangle) of a usersetting an input region on a palm 910 that is 15 cm away from thewearable device 100, by using an input tool, such as a finger, a pen, astylus or a stick.

When a distance between the palm 910 and the wearable device 100 is 15cm, a size of the palm 910 observed through the optical display 121 maybe smaller than that of the palm 810 of FIG. 8 that is 10 cm away fromthe wearable device 100. Accordingly, the wearable device 100 maydisplay a handwriting input window 920 on the optical display 121 suchthat the handwriting input window 920 matches the palm 910 observedthrough the optical display 121.

As shown in FIGS. 7 through 9, a virtual input interface is determinedin an order of the QWERTY keyboard 720, the mobile terminal keyboard820, and the handwriting input window 920 as a distance between a palm(an input region) and the wearable device 100 increases (as a firstdepth value of the input region increases), but an exemplary embodimentis not limited thereto. A virtual input interface may be determined inan order of the handwriting input window 920, the mobile terminalkeyboard 820, and the QWERTY keyboard 720 as the distance between thepalm (the input region) and the wearable device 100 decreases (as thefirst depth value of the input region decreases), and any type ofvirtual input interface may be determined.

FIGS. 10A and 10B are diagrams describing a type of a virtual inputinterface being adaptively changed according to a change of a depthvalue of an actual object where an input region is set, according toexemplary embodiments.

Referring to FIG. 10A, the wearable device 100 may recognize a gesture(for example, a gesture of drawing a rectangle) of a user setting aninput region on a palm 1010 that is 7 cm away from the wearable device100, by using an input tool, such as a finger, a pen, a stylus or astick. The wearable device 100 may display a QWERTY keyboard 1020 on theoptical display 121 such that the QWERTY keyboard 1020 matches the palm1010 observed through the optical display 121, based on the gesture.

While the QWERTY keyboard 1020 is displayed, the user may move the palm1010 away from the wearable device 100 such that a distance between thewearable device 100 and the palm 1010 is 10 cm.

When the distance between the wearable device 100 and the palm 1010 is10 cm, a size of the palm 1010 observed through the optical display 121may be smaller than that of the palm 1010 that is 7 cm away from thewearable device 100 as shown in FIG. 10A. Accordingly, the wearabledevice 100 may display a mobile terminal keyboard 1030, such as aCheonjiin keyboard, on the optical display 121 instead of the QWERTYkeyboard 1020 that was previously displayed. Thereby, the mobileterminal keyboard 1030 matches the palm 1010 observed through theoptical display 121.

Alternatively, the wearable device 100 may recognize a gesture (forexample, a gesture of drawing a rectangle) setting an input region onthe palm 1010 that is 10 cm away from the wearable device 100, by usingthe input tool. The wearable device 100 may display the mobile terminalkeyboard 1030 to overlap the palm 1010 based on the gesture.

While the mobile terminal keyboard 1030 is displayed, the user may movethe palm 1010 closer to the wearable device 100 such that the distancebetween the wearable device 100 and the palm 1010 is 7 cm.

When the distance between the wearable device 100 and the palm 1010 is 7cm, the size of the palm 1010 observed through the optical display 121may be larger than that of the palm 1010 that is 10 cm away from thewearable device 100. Accordingly, the wearable device 100 may displaythe QWERTY keyboard 1020 on the optical display 121 instead of themobile terminal keyboard 1030 that was displayed, such that the QWERTYkeyboard 1020 matches the palm 1010 observed through the optical display121.

As such, a user may change a type of a virtual input interface bychanging a location of an actual object (a distance between the actualobject and a wearable device) after an input region is set on the actualobject.

Referring to FIG. 10B, the wearable device 100 may obtain a firstdistance (for example, 7 cm) between the wearable device 100 and thepalm 1010 (an actual object), and display a first virtual inputinterface (for example, the QWERTY keyboard 1020) based on the firstdistance on the palm 1010 observed through the optical display 121. Forexample, the variable lens 240 a of FIG. 1B may be changed (or acurvature of a refracting surface of a variable lens may be changed) toadjust an incident angle of a first light 1025 incident on eyes of theuser, such that a distance from the eyes of the user to the QWERTYkeyboard 1020 recognized by the user is the first distance.

Also, the wearable device 100 may obtain a second distance (for example,10 cm) between the wearable device 100 and the palm 1010 (the actualobject), and display a second virtual input interface (for example, themobile terminal keyboard 1030) having the second distance on the palm1010 observed through the optical display 121. For example, the variablelens 240 a of FIG. 1B may be changed (or a curvature of a refractingsurface of a variable lens may be changed) to adjust an incident angleof a first light 1035 incident on eyes of the user, such that a distancefrom the eyes of the user to the mobile terminal keyboard 1030recognized by the user is the second distance.

FIGS. 10C and 10D are diagrams describing a type of a virtual inputinterface being changed based on a user input, according to exemplaryembodiments.

Referring to FIG. 10C, the wearable device 100 may display the firstvirtual input interface, for example, the QWERTY keyboard 1020, on theoptical display 121 such that the QWERTY keyboard 1020 matches the palm1010 observed through the optical display 121 based on a gesture of auser. Here, the wearable device 100 may display a key 1050 for changinga virtual input interface. When an input of selecting the key 1050 isreceived from the user, the wearable device 100 may display the secondvirtual input interface, for example, the mobile terminal keyboard 1030,in a region where the first virtual input interface was displayed, asshown in FIG. 10D. Also, the key 1050 for changing a virtual inputinterface may be displayed. Upon receiving the input of selecting thekey 1050 from the user, the wearable device 100 may display a thirdvirtual input interface in a region where the second virtual inputinterface was displayed, or may display the QWERTY keyboard 1020 asshown in FIG. 10C.

FIG. 11 is a flowchart of a method of providing a virtual inputinterface determined based on a size of an input region or a settingmotion of the input region, according to an exemplary embodiment.

Referring to FIG. 11, the wearable device 100 may set an input region byusing a user gesture for assigning a region for displaying a virtualinput interface, in operation S1110. Because operation S1110 has beendescribed in detail with reference to operation S210 of FIG. 2, andFIGS. 3A through 5B, details thereof are not repeated.

The wearable device 100 may determine a shape or type of a virtual inputinterface based on a size of the input region or the user gesture, inoperation S1120.

For example, when an area of the input region is equal to or less than afirst threshold value, the wearable device 100 may provide a virtualinput interface having a first area.

Alternatively, when the area of the input region is greater than thefirst threshold value and equal to or less than a second threshold valuethat is greater than the first threshold value, the wearable device 100may provide a virtual input interface having a second area that islarger than the first area. Here, a size of the input region may bedetermined by a height, width, diagonal length, or diameter, as well asthe area.

Also, the wearable device 100 may provide a different type of virtualinput interface based on a figure drawn by the user. The figure may bedrawn in the air or on an actual object and may be used to set the inputregion.

For example, when the user draws a first figure to set the input region,the wearable device 100 may recognize the first figure and provide avirtual input interface corresponding to the first figure. Also, whenthe user draws a second figure to set the input region, the wearabledevice 100 may provide a virtual input interface corresponding to thesecond figure.

This will be described in detail later with reference to FIGS. 12Athrough 15B.

Referring back to FIG. 11, in operation S1130 the wearable device 100may display the virtual input interface determined in operation S1120 onthe optical display 121 according to a size of the input region set inoperation S1110.

For example, the virtual input interface may be displayed on the opticaldisplay 121 such that the virtual input interface is shown in the inputregion. At this time, a shape of the virtual input interface may be thesame as a shape of the input region, and a size of the virtual inputinterface may be the same as or smaller than a size of the input region.

Also, in operation S1140 the wearable device 100 may obtain a firstdepth value of the input region and a second depth value of an inputtool touching or approaching the virtual input interface, and determinewhether an input is generated through the virtual input interface bycomparing the first and second depth values in operation S1150.

Because operations S1130 through S1150 of FIG. 11 have been describedabove with reference to FIGS. S230 through S250 of FIG. 2, detailsthereof are not repeated.

FIGS. 12A through 13B are diagrams describing types of virtual inputinterfaces being displayed according to dimensions of input regions.

As shown in FIG. 12A, the user of the wearable device 100 may draw afigure for setting an input region on a desk 1210. For example, the usermay use both hands to draw a rectangle 1220 having a first size (forexample, 20 cm×10 cm) on the desk 1210. Here, the wearable device 100may set an input region by using a gesture of the user drawing therectangle 1220 by using both hands.

Also, as shown in FIG. 12B, in response to the gesture drawing therectangle 1220, the wearable device 100 may display a virtual planokeyboard 1230 to overlap a region of the rectangle 1220 observed throughthe optical display 121. The wearable device 100 may display the virtualplano keyboard 1230 on the optical display 121 such that the virtualplano keyboard 1230 matches the first size of the rectangle 1220. Here,a size of the virtual plano keyboard 1230 may be determined according tothe first size of the rectangle 1220.

As shown in FIG. 13A, the user may draw a figure for setting an inputregion on a desk 1310. For example, the user may draw a rectangle 1320having a second size (for example, 10 cm×10 cm) on the desk 1310 byusing both hands. Here, the wearable device 100 may recognize a gestureof the user drawing the rectangle 1320 by using both hands as a gestureof setting an input region.

Also, as shown in FIG. 12B, in response to the gesture of drawing therectangle 1320, the wearable device 100 may display a virtual planokeyboard 1330 to overlap a region of the rectangle 1320 observed throughthe optical display 121. The wearable device 100 may display the virtualplano keyboard 1330 on the optical display 121 such that the virtualplano keyboard 1330 matches the second size of the rectangle 1320. Here,a size of the virtual plano keyboard 1330 may be determined according tothe second size of the rectangle 1320.

Alternatively, the wearable device 100 may provide a virtual inputinterface not only having different dimensions, but also differentshapes, based on a size of an input region.

Referring to FIGS. 12B and 13B, the virtual plano keyboard 1230 shown inFIG. 12B may be a plano keyboard displayed in one line, and the virtualplano keyboard 1330 shown in FIG. 13B may be a plano keyboard displayedin two lines, but are not limited thereto.

FIGS. 14A through 15B are diagrams describing types of virtual inputinterfaces being changed according to gestures setting input regions.

As shown in FIG. 14A, when the user draws a rectangle 1430 on a palm1410 observed through the optical display 121 by using a finger 1420,the wearable device 100 may recognize a gesture of drawing the rectangle1430 by using the image sensor 111, and set a region corresponding tothe rectangle 1430 as an input region.

At this time, as shown in FIG. 14B, the wearable device 100 may displaya virtual mobile terminal keyboard 1450 on the optical display 121 suchthat the virtual mobile terminal keyboard 1450 overlaps a rectangularregion observed through the optical display 121. For example, thewearable device 100 may display the virtual mobile terminal keyboard 145on the optical display 121 according to a size of the rectangularregion. Alternatively, the wearable device 100 may display the virtualmobile terminal keyboard 1450 on an opaque display.

As shown in FIG. 15A, when the user draws a circle 1530 on a palm 1510observed through the optical display 121 by using a finger 1520, thewearable device 100 may recognize a gesture of drawing the circle 1530by using the image sensor 111, and set a region corresponding to thecircle 1530 as an input region.

At this time, as shown in FIG. 15B, the wearable device 100 may displaya virtual dial pad 1550 on the optical display 121 such that the virtualdial pad overlaps a circular region observed through the optical display121. For example, the wearable device 100 may display the virtual dialpad 1550 on the optical display 121 to match a size of the circularregion. Alternatively, the wearable device 100 may display the virtualdial pad on an opaque display.

As such, the wearable device 100, according to an exemplary embodiment,may provide a virtual input interface having different shapes accordingto types of a gesture setting an input region, and information abouttypes, dimensions, and shapes of a virtual input interface, providedaccording to types of a gesture, may be stored in the wearable device100.

FIGS. 16A and 16B are diagrams describing providing a virtual inputinterface, determined based on an object where an input region is set,according to an exemplary embodiment.

Referring to FIG. 16A, the user may draw a figure (for example, arectangle) for setting an input region on a desk 1610 observed throughthe optical display 121. For example, the user may draw the rectangle onthe desk 1610 by using both hands.

The wearable device 100 may recognize a gesture of drawing the rectangleas a gesture of setting an input region, and set a region correspondingto the rectangle drawn on the desk 1610 as an input region.

Here, when the desk 1610 is an actual object where the input region isset, the user is able to use both hands, and thus the wearable device100 may determine a QWERTY keyboard 1620 as a virtual input interface.

Also, the wearable device 100 may display the QWERTY keyboard 1620 onthe optical display 121 such that the QWERTY keyboard 1620 overlaps arectangular region of the desk 1610 observed through the optical display121. For example, the wearable device 100 may display the QWERTYkeyboard 1620 on the optical display 121 according to a size of therectangular region. Alternatively, the wearable device 100 may displaythe QWERTY keyboard 1620 on an opaque display.

Referring to FIG. 16B, the user may draw a figure (for example, arectangle) for setting an input region on a palm 1630 observed throughthe optical display 121. For example, the user may draw the rectangle onthe palm 1630 by using a finger.

The wearable device 100 may recognize a gesture of drawing the rectangleas a gesture of setting an input region, and set a region correspondingto the rectangle drawn on the palm 1630 as the input region.

Here, when the palm 1630 is an actual object where the input region isset, the user is able to use only one hand, and thus the wearable device100 may set a mobile terminal keyboard 1640 as a virtual inputinterface.

Also, the wearable device 100 may display the mobile terminal keyboard1640 on the optical display 121 to overlap the rectangular region on thepalm 1630 observed through the optical display 121. For example, thewearable device 100 may display the mobile terminal keyboard 1640 on theoptical display 121 according to a size of the rectangular region.Alternatively, the wearable device 100 may display the mobile terminalkeyboard 1640 on an opaque display.

A color of a virtual input interface may be determined according to acolor of an input region. For example, when the color of the inputregion is a first color, the color of the virtual input interface may bedetermined to be a second color that is different from the first color,or a third color that is a complementary color of the first color. Assuch, the user may be able to easily distinguish the virtual inputinterface overlapping the input region observed through the opticaldisplay 121, from the input region.

FIGS. 17A through 17C are diagrams describing a virtual input interfaceprovided by the wearable device 100, the virtual input interfacedetermined based on a type of an actual object where an input region isset, according to an exemplary embodiment.

As shown in FIGS. 17A through 17C, it is assumed that the user wearingthe wearable device 100 performs a gesture of setting an input region ona book 1700 while reading the book 1700.

The wearable device 100, according to an exemplary embodiment, mayrecognize a type of an actual object where an input region is set, byusing the image sensor 111. For example, as shown in FIG. 17A, thewearable device 100 may detect a gesture of the user drawing a rectangle1710 on the book 1700 by using an input tool 1701, by using the imagesensor 111. At this time, the wearable device 100 may identify that thebook 1700 is an actual object on which the input region is drawn via animage process, and accordingly, determine a notepad as a virtual inputinterface corresponding to the book 1700.

As shown in FIG. 17B, the wearable device 100 may display a virtualnotepad 1720 on the optical display 121 such that the virtual notepad1720 overlaps the input region set on the book 1700 observed through theoptical display 121.

Alternatively, the wearable device 100, according to an exemplaryembodiment, may set a blank space of the book 1700, in which text orimages are not displayed, as the input region via an image process, anddisplay the virtual notepad 1720 on the optical display 121 such thatthe virtual notepad 1720 overlaps the blank space observed through theoptical display 121.

Also, the wearable device 100 may obtain a first depth value of the book1700 and a second depth value of the input tool 1701, and display aninput on the virtual notepad 1720 when it is determined that the inputis generated based on the first and second depth values.

Also, as shown in FIG. 17C, the wearable device 100 may store input data1730 displayed on a virtual notepad based on a user input.

As such, when the user reads the book 1700 while wearing the wearabledevice 100, the user may easily store important information by using avirtual notepad.

FIGS. 18A and 18B are diagrams describing a virtual input interface, thevirtual input interface determined based on an input tool that sets aninput region, according to an exemplary embodiment.

Referring to FIGS. 18A and 18B, the user may draw a figure (for example,a rectangle) for setting an input region in the air or on an actualobject, by using an input tool, such as a finger or a pen.

The wearable device 100 may recognize a gesture of drawing the rectangleby using the input tool, as a gesture of setting an input region, andset the rectangle drawn in the air or on the actual object as the inputregion.

When the input region is set, the wearable device 100 may determine avirtual input interface based on the input tool setting the inputregion.

For example, as shown in FIG. 18A, when a finger 1820 is used as aninput tool to set an input region 1810, the wearable device 100 maydetermine a mobile terminal keyboard 1830 that is easily touched by thefinger 1820 as a virtual input interface.

As such, the wearable device 100 may display the mobile terminalkeyboard 1830 on the optical display 121 to overlap the input region1810 observed through the optical display 121. Alternatively, thewearable device 100 may display the mobile terminal keyboard 1830 on anopaque display.

Meanwhile, as shown in FIG. 18B, when a pen 1850 is used as an inputtool to set an input region 1840, the wearable device 100 may determinea handwriting input window 1860 easily used by the pen 1850 as a virtualinput interface.

As such, the wearable device 100 may display the handwriting inputwindow 1860 on the optical display 121 to overlap the input region 1840observed through the optical display 121. Alternatively, the wearabledevice 100 may display the handwriting input window 1860 on an opaquedisplay.

FIG. 19 is a flowchart illustrating a method of providing a virtualinput interface determined based on an application being executed by thewearable device 100, according to an exemplary embodiment.

Referring to FIG. 19, the wearable device 100 may execute an applicationin operation S1910. For example, the wearable device 100 may select andexecute any one of a plurality of applications provided in the wearabledevice 100. Here, the user may execute an application by using a voiceinput or a key input.

For example, when a message is to be transmitted to an external device,the wearable device 100 may execute a message application. At this time,the message may be a text message, an instant message, a chattingmessage, or an email.

Alternatively, the wearable device 100 may receive a message from theexternal device, and execute the message application in order to respondto or view the received message.

When an application that requires input of text or numbers, such as themessage application, is executed (when a virtual input interface is tobe displayed), the wearable device 100 may receive a gesture and, inoperation S1920, set an input region based on the gesture. Becauseoperation S1920 has been described above in detail with reference tooperation S210 of FIG. 2 and FIGS. 3A through 5B, details thereof arenot repeated.

In operation S1930, the wearable device 100 may determine a virtualinput interface based on a type of the application being executed.

For example, when the message application is executed and a text inputis required to prepare a message, the wearable device 100 may determinea virtual keyboard, such as a QWERTY keyboard or a mobile terminalkeyboard, as a virtual input interface. Alternatively, when the messageapplication requires a numerical input, such as a phone number of areceiver, the wearable device 100 may determine a virtual dial pad as avirtual input interface, as will be described in detail later withreference to FIGS. 20A and 20B.

The wearable device 100 may display the virtual input interface tooverlap the input region, in operation S1940.

Here, the wearable device 100 may display the virtual input interface inthe form of AR, MR, or VR.

For example, when the wearable device 100 displays the virtual inputinterface in the form of AR or MR, the virtual input interface may bedisplayed on a transparent display to overlap the input region.

Alternatively, when the wearable device 100 displays the virtual inputinterface in the form of VR, the virtual input interface may bedisplayed on an opaque display to overlap the input region.

The wearable device 100 may obtain a first depth value of the inputregion and a second depth value of the input tool touching the virtualinput interface, in operation S1950.

The wearable device 100 may determine whether an input is generatedthrough the virtual input interface by comparing the first and seconddepth values, in operation S1960.

Because operations S1940 through S1960 of FIG. 19 correspond tooperations S230 through S250 of FIG. 2, details thereof are notrepeated.

FIGS. 20A and 20B are diagrams describing providing a virtual inputinterface determined based on a type of an application being executed,according to an exemplary embodiment.

The wearable device 100 may execute a call application based on a userinput. For example, the call application may be executed by using avoice input or a key input.

When the call application is executed, the user may set an input regionto display a virtual input interface to input a phone number of a personthe user wants to call. For example, the wearable device 100 mayrecognize a gesture of the user drawing the input region on a palm 2010,and set the input region on the palm 2010.

Then, the wearable device 100 may determine a virtual input interfacecorresponding to the call application being executed, and as shown inFIG. 20A, display a virtual dial pad 2020 that is the virtual inputinterface on the optical display 121 such that the virtual dial pad 2020overlaps the palm 2010 observed through the optical display 121.

Alternatively, the wearable device 100 may execute a notepad applicationbased on a user input. For example, the user may execute the notepadapplication by using a voice input or a key input.

When the notepad application is executed, the user may set an inputregion to display a virtual input interface for inputting text. Forexample, the wearable device 100 may recognize a gesture of setting theinput region on the palm 2010, and set the input region on the palm2010.

Then, the wearable device 100 may determine a virtual input interfacecorresponding to the notepad application, and as shown in FIG. 20B,display a virtual mobile terminal keyboard 2030 that is the virtualinput interface on the optical display 121, such that the virtual mobileterminal keyboard 2030 overlaps the palm 2010 observed through theoptical display 121. However, an exemplary embodiment is not limitedthereto.

FIG. 21 is a diagram describing a virtual input interface determinedbased on a type of content being executed, according to an exemplaryembodiment.

The wearable device 100, according to an exemplary embodiment, maydetermine a virtual input interface to be displayed based on a type ofcontent being executed by the wearable device 100.

Examples of the content include a still image, a moving image, text, anda webpage, but are not limited thereto. For example, the content may beeducational content, movie content, broadcast content, game content,commercial content, picture content, or news content.

Executing of content may mean that the content is displayed, output, orreproduced.

Referring to FIG. 21, the wearable device 100 may detect a gesturesetting an input region while executing game content 2110. At this time,the wearable device 100 may display a virtual game control panel 2115corresponding to the game content 2110 on a transparent or opaquedisplay to overlap the input region.

Alternatively, the wearable device 100 may detect a gesture setting aninput region while executing music content 2120, such as drum playingcontent. At this time, the wearable device 100 may display a drumplaying panel 2125 corresponding to the music content 2120 on atransparent or opaque display to overlap the input region.

Alternatively, the wearable device 100 may detect a gesture setting aninput region while displaying a webpage 2130. At this time, the wearabledevice 100 may display a virtual keyboard 2135 to search for informationfrom the webpage 2130 on a transparent or opaque display to overlap theinput region.

FIGS. 22A through 23B are diagrams describing virtual input interfacesthat are same as previous virtual input interfaces provided when thewearable device 100 recognizes actual objects where the previous virtualinput interfaces were provided, according to exemplary embodiments.

As shown in FIG. 22A, when the user draws a rectangle 2230 on a palm2210 by using a finger 2220, the wearable device 100 may recognize agesture of drawing the rectangle 2230 by using the image sensor 111, andset a region corresponding to the rectangle 2230 as an input region.

Here, the wearable device 100 may determine a type of a virtual inputinterface to be displayed, based on a type of an application currentlybeing executed. For example, when a notepad application that requires atext input is being executed, the wearable device 100 may determine amobile terminal keyboard 2250 as the virtual input interface, but anexemplary embodiment is not limited thereto.

As shown in FIG. 22B, the wearable device 100 may display the mobileterminal keyboard 2250 on the optical display 121 such that the mobileterminal keyboard 2250 overlaps a rectangular region observed throughthe optical display 121. Alternatively, the wearable device 100 maydisplay the mobile terminal keyboard 2250 on an opaque display.

Then, the wearable device 100 may recognize an object that is the sameas an actual object (the palm 2210 of FIG. 22B) on which the virtualinput interface was provided, while executing the notepad application.

For example, as shown in FIG. 23A, the wearable device 100 may detectthe palm 2210 of the user by using the image sensor 111. At this time,the wearable device 100 may identify that the palm 2210 is the actualobject (the palm 2210 of FIG. 22B) on which the virtual input interfacewas provided, via an image process.

When the actual object is recognized, the wearable device 100 mayprovide a virtual input interface the same as that previously providedin the input region, as shown in FIG. 23B.

For example, the wearable device 100 may display the mobile terminalkeyboard 2250 previously provided on the optical display 121 to overlapan input region 2270 observed through the optical display 121, even whenthe user does not draw a rectangle by using an input tool to set aninput region.

As such, the user may enable the wearable device 100 to recognize anactual object where a virtual input interface was displayed such thatthe wearable device 100 provides a virtual input interface that wasprovided.

FIG. 24 is a flowchart illustrating a method of providing a virtualinput interface in an input region set in the air, according to anexemplary embodiment.

Referring to FIG. 24, the wearable device 100 may set an input region inthe air in operation S2410. For example, as described above withreference to FIG. 3A, the wearable device 100 may recognize a figuredrawn by the user in the air by using an input tool, such as a finger, apen, a stylus or a stick, and set a region corresponding to the figureas an input region.

The wearable device 100 may determine a virtual input interface inoperation S2420.

For example, the wearable device 100 may determine the virtual inputinterface based on attributes of the input region. The wearable device100 may determine the virtual input interface to be displayed on theoptical display 121 based on at least one of a size of the input region,a shape of the input region, a distance between the input region and thewearable device 100 (a first depth value of the input region), and agesture of setting the input region.

Alternatively, the wearable device 100 may determine the virtual inputinterface based on a type of application or content being executed. Forexample, when the application being executed requires a text input, thewearable device 100 may determine a virtual keyboard, such as a QWERTYkeyboard or a mobile terminal keyboard, as the virtual input interface.Alternatively, when the application being executed requires a numericalinput, the wearable device 100 may determine a virtual dial pad as thevirtual input interface.

In operation S2430, wearable device 100 may display the virtual inputinterface to overlap the input region.

At this time, the wearable device 100 may display the virtual inputinterface in the form of AR, MR, or VR.

For example, when the virtual input interface is displayed in the formof AR or MR, the wearable device 100 may display the virtual inputinterface on a transparent display such that the virtual input interfaceoverlaps the input region (a 2D or 3D space of the real world) observedthrough the transparent display.

Alternatively, when the virtual input interface is displayed in the formof VR, the wearable device 100 may capture a first image (an actualimage) including the input region (the 2D or 3D space of the realworld), and generate a second image by adding the virtual inputinterface (a virtual image) to the input region of the first image. Thewearable device 100 may display the second image in which the virtualinput interface overlaps the input region on an opaque display.

The wearable device 100 may obtain a first depth value of the inputregion and a second depth value of the input tool touching the virtualinput interface, in operation S2440.

The wearable device 100 may measure a distance (a depth value of theinput region, i.e., the first depth value) from the wearable device 100to the input region by using the depth sensor 112.

For example, when the input region is set in the air, the wearabledevice 100 may obtain the first depth value of the input region bymeasuring a depth value of the input tool setting the input region inthe air.

Meanwhile, if the input region is on an uneven surface and the inputregion does not exist on the same plane, there may be a plurality ofdepth values of the input region. When there are the plurality of depthvalues of the input region, the first depth value may be one of anaverage depth value of the plurality of depth values, a minimum depthvalue of the plurality of depth values, or a maximum depth value of theplurality of depth values, but is not limited thereto.

Also, the wearable device 100 may measure a distance (a depth value ofthe input tool, i.e., the second depth value) from the wearable device100 to the input tool touching the virtual input interface, by using thedepth sensor 112.

When the input tool is a 3D object, there may be a plurality of depthvalues of the input tool. When there are a plurality of values of theinput tool, the second depth value may be one of an average depth valueof the plurality of depth values, a minimum depth value of the pluralityof depth values, or a maximum depth value of the plurality of depthvalues, but is not limited thereto.

For example, when the virtual input interface is touched by using theinput tool, a point (an end point of the input tool) where the inputtool and the virtual input interface contact each other may be thesecond depth value.

Also, the wearable device 100 may track the input tool that is moving inreal-time by using the depth sensor 112, and calculate the second depthvalue that changes in real-time.

The wearable device 100 may compare the first and second depth values inoperation S2450.

For example, the wearable device 100 may determine whether the seconddepth value is greater than the first depth value, and when it isdetermined that the second depth value is greater than the first depthvalue, determine that an input is generated through the virtual inputinterface, in operation S2460.

However, when it is determined that the second depth value is smallerthan the first depth value, the wearable device 100 may determine thatan input is not generated through the virtual input interface, inoperation S2470.

The determining of whether an input is generated will now be describedin detail with reference to FIGS. 25A and 25B.

FIGS. 25A and 25B are diagrams describing a method of determiningwhether an input is generated through a virtual input interface, when aninput region is set in the air.

Referring to FIGS. 25A and 25B, the wearable device 100 may display avirtual keyboard 2510 on a transparent or opaque display such that thevirtual keyboard 2510 overlaps an input region set in the air.

The wearable device 100 may also measure a first depth value of thevirtual keyboard 2510 by using the depth sensor 112.

Meanwhile, even when the user wearing the wearable device 100 moves, thewearable device 100 is able to display the virtual keyboard 2510 on thetransparent or opaque display such that the virtual keyboard 2510continuously overlaps the input region having the first depth value. Forexample, even when the user is walking, the wearable device 100 mayadjust the virtual keyboard 2510 to be continuously displayed in aregion away from the wearable device 100 by a certain distance (thefirst depth value) by using the depth sensor 112.

Also, referring to FIGS. 25A and 25B, the user may input data bytouching the virtual keyboard 2510 in the air, by using a finger 2520.

Here, the wearable device 100 may determine whether an input isgenerated through the virtual keyboard 2510 by measuring a depth value(a second depth value) of the finger 2520 touching the virtual keyboard2510.

For example, as shown in FIG. 25A, the finger 2520 may approach thevirtual keyboard 2510 in order to select a button displayed on thevirtual keyboard 2510. At this time, when the finger 2520 does not passthrough the input region where the virtual keyboard 2510 is displayed,the second depth value of the finger 2520 may be smaller than the firstdepth value.

When the second depth value of the finger 2520 is smaller than the firstdepth value, the wearable device 100 may recognize that the user is nottouching the virtual keyboard 2510, and determine that an input is notgenerated through the virtual keyboard 2510.

On the other hand, as shown in FIG. 25B, when the finger 2520 passesthrough the input region where the virtual keyboard 2510 is displayed,the second depth value of the finger 2520 may be greater than the firstdepth value.

When the second depth value of the finger 2520 is greater than the firstdepth value, the wearable device 100 may recognize that the user istouching the virtual keyboard 2510.

When it is determined that the user is touching the virtual keyboard2510, the wearable device 100 may detect a location of the finger 2520on the virtual keyboard 2510 by using the image sensor 111. The wearabledevice 100 may determine input data of the user based on the detectedlocation of the finger 2520. For example, when the finger 2520 ispassing through an “enter” button on the virtual keyboard 2510, thewearable device 100 may determine that the user selected the “enter”button.

According to an exemplary embodiment, the wearable device 100 mayaccurately determine whether an input is generated through a virtualinput interface provided in the air by comparing a first depth value ofan input region set in the air to a second depth value of an input tool(for example, a finger or a pen) touching the virtual input interface.

FIG. 26 is a flowchart illustrating a method of providing a virtualinput interface in an input region set in the air or on an actualobject, according to an exemplary embodiment.

Referring to FIG. 26, the wearable device 100 may set an input region inthe air or on an actual object, in operation S2610. For example, asdescribed above with reference to FIG. 3, the wearable device 100 mayrecognize a figure drawn by the user in the air or on the actual object,such as a palm, a desk, or a wall, by using an input tool, such as afinger, a pen, a stylus or a stick, and set a region corresponding tothe figure as the input region.

Alternatively, as described above with reference to FIG. 4, the wearabledevice 100 may recognize a pre-set object, and set a regioncorresponding to the pre-set object as the input region.

Alternatively, as described above with reference to FIG. 5, the wearabledevice 100 may recognize an operation of the user touching a pre-setobject by using the input tool, and set a region corresponding to thetouched pre-set object as the input region.

The wearable device 100 may determine a virtual input interface, inoperation S2620.

For example, the wearable device 100 may determine the virtual inputinterface based on attributes of the input region. The wearable device100 may determine the virtual input interface to be displayed on theoptical display 121 based on at least one of a size of the input region,a shape of the input region, a distance (a first depth value of theinput region) between the input region and the wearable device 100, atype of the actual object where the input region is set, and a gestureof setting the input region.

Alternatively, the wearable device 100 may determine the virtual inputinterface based on a type of application or content being executed. Forexample, when the application being executed requires a text input, thewearable device 100 may determine a virtual keyboard, such as a QWERTYkeyboard or a mobile terminal keyboard, as the virtual input interface.Alternatively, when the application being executed requires a numericalinput, the wearable device 100 may determine a virtual dial pad as thevirtual input interface.

The wearable device 100 may display the virtual input interface tooverlap the input region, in operation S2630.

At this time, the wearable device 100 may display the virtual inputinterface in the form of AR, MR, or VR.

For example, when the wearable device 100 displays the virtual inputinterface in the form of AR or MR, the wearable device 100 may displaythe virtual input interface on a transparent display such that thevirtual input interface overlaps the input region.

Alternatively, when the wearable device 100 displays the virtual inputinterface in the form of VR, the wearable device 100 may display thevirtual input interface on an opaque display such that the virtual inputinterface overlaps the input region.

Because operation S2630 of FIG. 26 is the same as operation S2430 ofFIG. 24, details thereof are not repeated.

The wearable device 100 may obtain a first depth value of the inputregion and a second depth value of the input tool touching the virtualinput interface, in operation S2640.

For example, when the input region is set in the air, the wearabledevice 100 may obtain the first depth value of the input region bymeasuring a depth value of the input tool while setting the input regionin the air.

Alternatively, when the input region is set on an actual object, thewearable device 100 may obtain the first depth value of the input regionby measuring a depth value of the actual object (a distance from thewearable device 100 to the actual object).

Also, the wearable device 100 may measure a distance (a depth value ofthe input tool, i.e., the second depth value) from the wearable device100 to the input tool touching the virtual input interface by using thedepth sensor 112.

Also, the wearable device 100 may track the input tool that is moving inreal-time and calculate the second depth value in real-time, by usingthe depth sensor 112.

In operation S2650, the wearable device 100 may compare a differencebetween the first and second values with a threshold value.

For example, in operation S2660, the wearable device 100 may determinewhether the difference is smaller than the threshold value, and when itis determined that the difference is smaller than the threshold value,determine that an input is generated through the virtual inputinterface.

In operation S2670, when it is determined that the difference is equalto or greater than the threshold value, the wearable device 100 maydetermine that an input is not generated through the virtual inputinterface.

The determining of whether an input is generated will now be describedin detail with reference to FIGS. 27A and 27B.

FIGS. 27A and 27B are diagrams describing a method of determiningwhether an input is generated through a virtual input interface, when aninput region is set on an actual object.

Referring to FIGS. 27A and 27B, the wearable device 100 may display avirtual keyboard 2730 on the optical display 121 such that the virtualkeyboard 2730 overlaps an actual object, such as a palm 2710, observedthrough the optical display 121.

Also, the wearable device 100 may measure a first depth value of thepalm 2710 by using the depth sensor 112.

Meanwhile, the wearable device 100 may track the palm 2710 in real-time,even if a location of the palm 2710 changes, and continuously adjust thevirtual keyboard 2730 to overlap the palm 2710 observed through theoptical display 121 by continuously calculating the first depth value asit changes in real-time.

Also, referring to FIG. 27B, the user may input data by touching thevirtual keyboard 2730 shown on the palm 2710 by using a finger 2720.

At this time, the wearable device 100 may measure a depth value (asecond depth value) of the finger 2720 touching the virtual keyboard2730 to determine whether an input is generated through the virtualkeyboard 2730.

As shown in FIG. 27A, when the finger 2720 is away from the palm 2710 byat least a certain distance, the wearable device 100 may determine thatan input is not generated through the virtual keyboard 2730.

For example, when a difference between the first depth value of the palm2710 where the virtual keyboard 2730 is shown and the second depth valueof the finger 2720 is equal to or greater than a threshold value, it maybe determined that the user is not touching the virtual keyboard 2730,and that an input is not generated through the virtual keyboard 2730.

As shown in FIG. 27B, the user may approach the finger 2720 near thevirtual keyboard 2730 to select a button displayed on the virtualkeyboard 2730. Here, when the difference between the first and seconddepth values is smaller than the threshold value, it may be determinedthat the user is touching the virtual keyboard 2730.

Also, when the difference between the first and second depth values issmaller than the threshold value, the wearable device 100 may detect alocation of the finger 2720 on the virtual keyboard 2730 by using theimage sensor 111. The wearable device 100 may determine input data basedon the location of the finger 2720. For example, when the finger 2720passes through an “enter” button on the virtual keyboard 2730, thewearable device 100 may determine that the user selected the “enter”button.

According to an exemplary embodiment, the wearable device 100 mayaccurately determine whether an input is generated through a virtualinput interface provided in the air, or on an actual object, bycomparing a first depth value of an input region set by the user in theair or on the actual object and a second depth value of an input tool(for example, a finger or a pen) touching the virtual input interface.

FIGS. 28A and 28B are diagrams describing a method of obtaining a firstdepth value of an input region and a second depth value of an inputtool, according to an exemplary embodiment.

As shown in FIGS. 28A and 28B, it is assumed that a virtual keyboard isdisplayed by using a palm of a user as an input region, when a keyboardinput is required.

Referring to FIG. 28A, the user may set an input region on a left palm2820 while wearing the wearable device (the first wearable device) 100in a glasses type, and may be wearing a second wearable device 2810 on aleft wrist. Here, the second wearable device 2810 may be wearable on awrist of the user, like a watch, a bracelet, or a band, but is notlimited thereto.

The second wearable device 2810 may include a location sensor, and maysense location information of the second wearable device 2810 by usingthe location sensor. Also, the first and second wearable devices 100 and2810 may transmit and receive data to and from each other by including acommunicator, and the second wearable device 2810 may transmit thesensed location information of the second wearable device to the firstwearable device 100.

Meanwhile, the first wearable device 100 may include a location sensor,and may sense location information of the first wearable device 100 byusing the location sensor.

The first wearable device 100 may compare the sensed locationinformation of the first wearable device 100 with the received locationinformation of the second wearable device 2810 to calculate a distancebetween the first and second wearable devices 100 and 2810.

A distance between the left wrist wearing the second wearable device2810 and the first wearable device 100 may be similar to a distancebetween the left palm 2820, set as an input region where a virtualkeyboard 2840 is displayed, and the first wearable device 100.Accordingly, the first wearable device 100 may determine the distancebetween the first and second wearable devices 100 and 2810 as a firstdepth value.

As such, the first wearable device 100 may accurately obtain the firstdepth value by using the location information of the second wearabledevice 2810.

Also, the second wearable device 2810 may include a motion sensor, andrecognize a touch input by detecting motion, such as vibration,generated when the left palm 2820 is touched, by using the motionsensor. When the touch input is recognized, the second wearable device2810 may transmit data about the touch input to the first wearabledevice 100 through the communicator. Accordingly, the first wearabledevice 100 may accurately recognize that the touch input is generated byusing sensing information of the second wearable device 2810.

Meanwhile, referring to FIG. 28B, the user may set the input region onthe left palm 2820 while wearing the first wearable device 100 in aglasses type, and may be wearing a third wearable device 2850 on a rightfinger 2830. Here, the third wearable device 2850 may be wearable on afinger, like a thimble or a ring, but is not limited thereto.

The third wearable device 2850 may include a location sensor, and senselocation information of the third wearable device 2850 by using thelocation sensor.

Also, the first and third wearable devices 100 and 2850 may transmit andreceive data to and from each other by using an included communicator,and the third wearable device 2850 may transmit the sensed locationinformation of the third wearable device 2850 to the first wearabledevice 100.

The first wearable device 100 may include a location sensor, and maysense the location information of the first wearable device 100 by usingthe location sensor.

The first wearable device 100 may compare the sensed locationinformation of the first wearable device 100 with the received locationinformation of the third wearable device 2850 to calculate a distancebetween the first and third wearable devices 100 and 2850.

As shown in FIG. 28B, when the right finger 2830 wearing the thirdwearable device 2850, such as a thimble, is used as an input tooltouching the virtual keyboard 2840, a depth value of the third wearabledevice 2850 may be a depth value of the right finger 2830, and thedistance between the first and third wearable devices 100 and 2850 maybe determined as a second depth value.

As such, the first wearable device 100 may accurately obtain the seconddepth value by using the location information of the third wearabledevice 2850.

Also, the third wearable device 2850 may include a pressure sensor, andmay recognize a touch input by detecting pressure generated when theleft palm 2820 is touched, by using the pressure sensor. When the touchinput is recognized, the third wearable device 2850 may transmit dataabout the touch input to the first wearable device 100 through thecommunicator. As such, the first wearable device 100 may accuratelyrecognize whether the touch input is generated by using sensinginformation of the third wearable device 2850.

FIG. 29 is a flowchart illustrating a method of providing feedback aboutwhether an input is generated through a virtual input interface,according to an exemplary embodiment.

Referring to FIG. 29, the wearable device 100 may set an input region,in operation S2910.

When the input region is set, the wearable device 100 may determine avirtual input interface in operation S2920.

For example, the wearable device 100 may determine the virtual inputinterface based on attributes of the input region. The wearable device100 may determine the virtual input interface to be displayed on theoptical display 121 based on at least one of a size of the input region,a shape of the input region, a distance (a first depth value of theinput region) between the input region and the wearable device 100, anda gesture of setting the input region.

Alternatively, the wearable device 100 may determine the virtual inputinterface based on a type of application or content being executed. Forexample, when an application being executed requires a text input, thewearable device 100 may determine a virtual keyboard, such as a QWERTYkeyboard or a mobile terminal keyboard, as the virtual input interface.Alternatively, when an application being executed requires a numericalinput, the wearable device 100 may determine a virtual dial pad as thevirtual input interface.

In operation S2930, the wearable device 100 may display the virtualinput interface to overlap the input region.

At this time, the wearable device 100 may display the virtual inputinterface in the form of AR, MR, or VR.

For example, when the wearable device 100 displays the virtual inputinterface in the form of AR or MR, the virtual input interface may bedisplayed on a transparent display to overlap the input region.

Alternatively, when the wearable device 100 displays the virtual inputinterface in the form of VR, the virtual input interface may bedisplayed on an opaque display to overlap the input region.

The wearable device 100 may obtain the first depth value of the inputregion and a second depth value of an input tool touching the virtualinput interface, in operation S2940.

The wearable device 100 may determine whether an input is generatedthrough the virtual input interface by comparing the first and seconddepth values, in operation S2950.

Because operations S2930 through S2950 of FIG. 29 correspond tooperations S230 through S250 of FIG. 2, additional details thereof arenot repeated.

In operation S2960, when it is determined that an input is generatedthrough the virtual input interface, the wearable device 100 may outputa notification signal corresponding to the generated input. Examples ofthe notification signal include a video signal, an audio signal, and ahaptic signal, but are not limited thereto.

The outputting of the notification signal will be described in detailwith reference to FIGS. 30 through 32.

FIGS. 30 and 31 are diagrams describing outputting notification signalscorresponding to whether inputs are generated by wearable devices,according to exemplary embodiments.

As shown in FIGS. 30 and 31, the wearable device 100 may recognize agesture setting an input region on a palm 3010, and display a virtualkeyboard 3030 on the optical display 121 to overlap the palm 3010observed through the optical display 121.

At this time, the user may touch a button displayed on the virtualkeyboard 3030 by using a finger 3020 to generate an input.

The wearable device 100 may compare a depth value (a second depth value)of the finger 3020 with a depth value (a first depth value) of thefinger 3010 and determine that the input is generated by the finger 3020when a difference between the first and second depth values is smallerthan a threshold value.

When the input is generated, the wearable device 100 may detect alocation of the finger 3020 on the virtual keyboard 3030, and generateinput data about a button 3040 at the location of the finger 3020. Also,the wearable device 100 may provide feedback to the user such that theuser easily recognizes the input.

For example, a color of the button 3040 may be changed. Alternatively,when the input is generated through the virtual keyboard 3030, an alarmmay be output.

Alternatively, when an input is generated through a virtual inputinterface, the wearable device 100 may output a haptic signal by using aperipheral device.

As shown in FIG. 31, the user may be wearing a second wearable device3150 on the finger 3020 touching the virtual keyboard 3030. Here, thesecond wearable device 3150 may be wearable on the finger 3020, such asa thimble or a ring, but is not limited thereto, as long as the secondwearable device 3150 is wearable.

Also, the second wearable device 3150 may include a haptic module. Thehaptic module may generate various tactile effects. An example of thetactile effect generated by the haptic module includes a vibrationeffect. When the haptic module generates vibration as a tactile effect,strength and pattern of the vibration may be changed, and differenttypes of vibration may be output, either in combination or sequentially.

When an input is generated on a button displayed on the virtual keyboard3030, the wearable device 100 may request the second wearable device3150 to output a haptic signal through a communicator.

Then, in response, the second wearable device 3150 may output a hapticsignal through the haptic module.

FIG. 32 is a diagram describing outputting of a notification signalcorresponding to whether an input is generated through a virtual inputinterface, according to an exemplary embodiment.

As shown in FIG. 32, the wearable device 100 may recognize a gesture ofa user setting an input region on a desk 3210, and display a virtualplano keyboard 3220 on a transparent or opaque display to overlap thedesk 3210.

At this time, the user may generate an input by touching the virtualplano keyboard 3220 by using a finger 3230.

The wearable device 100 may compare a depth value (a second depth value)of the finger 3230 with a depth value (a first depth value) of the desk3210, and determine that an input is generated by the finger 3230 when adifference between the first and second depth values is smaller than athreshold value.

When it is determined that an input is generated, the wearable device100 may detect a location of the finger 3230 on the virtual planokeyboard 3220, and display a virtual image 3250 on the virtual planokeyboard 3220 at the location of the finger 3230. As such, the user mayeasily recognize that the input is generated at a location where thevirtual image 3250 is displayed.

FIGS. 33 and 34 are block diagrams of the wearable device 100 accordingto exemplary embodiments.

As shown in FIG. 33, the wearable device 100, according to an exemplaryembodiment, may include a sensor 110, the optical display 121, and acontroller 130. However, components shown in FIG. 33 are not allessential. The wearable device 100 may include more or less componentsthan those shown in FIG. 33.

For example, as shown in FIG. 34, the wearable device 100, according toan exemplary embodiment, may further include a user input 140, acommunicator 150, and a memory 160, as well as the sensor 110, anoutputter 120, and the controller 130.

The above components will now be described in detail.

The sensor 110 may detect a state of the wearable device 100 or a statearound the wearable device 100, and transmit information about thedetected state to the controller 130.

The sensor 110 may include the image sensor 111 and the depth sensor112. The wearable device 100 may obtain an image frame of a still imageor a moving image through the image sensor 111. Here, an image capturedthrough the image sensor 111 may be processed by the controller 130 or aseparate image processor.

According to an exemplary embodiment, the image sensor 111 may recognizea gesture of setting an input region in the air or on an actual object.For example, the image sensor 111 may recognize a gesture of setting aninput region in the air, or on an actual object, by using an input tool.

Alternatively, the image sensor 111 may recognize a pre-set object to beset as an input region, and recognize a gesture of touching the pre-setobject by using an input tool. Alternatively, the image sensor 111 maycapture a first image including an input region.

According to an exemplary embodiment, the depth sensor 112 may obtain afirst depth value of an input region and a second depth value of aninput tool touching a virtual input interface. For example, the depthsensor 112 may measure a distance from the wearable device 100 to aninput region and a distance from the wearable device 100 to an inputtool.

Alternatively, when an input region is set on an actual object, thedepth sensor 112 may measure a distance from the wearable device 100 tothe actual object, and obtain a first depth value of an input region byusing the measured distance.

According to an exemplary embodiment, the sensor 110 may include atleast one of an acceleration sensor 113, a location sensor 114, such asa global positioning system (GPS), an atmospheric pressure sensor 115, atemperature/humidity sensor 116, a terrestrial magnetic sensor 117, agyroscope sensor 118, and a microphone 119, as well as the image sensor111 and the depth sensor 112.

The microphone 119 receives an external sound signal and processes theexternal audio signal to electric voice data. For example, themicrophone 119 may receive the external sound signal from an externaldevice or a person. The microphone 119 may use any one of various noiseremoving algorithms to remove noise generated while receiving theexternal sound signal.

Because functions of the acceleration sensor 113, the location sensor114, the atmospheric pressure sensor 115, the temperature/humiditysensor 116, the terrestrial magnetic sensor 117, and the gyroscopesensor 118 are intuitively inferable by one of ordinary skill in theart, details thereof are not provided here.

The outputter 120 may output an audio signal, a video signal, or avibration signal, and include the optical display 121, a sound output122, and a vibration motor 123.

The optical display 121 may display information processed by thewearable device 100. For example, the optical display 121 may display auser interface (UI) or a graphical user interface (GUI) related to aphone call in a call mode, and display a virtual input interface in aninput mode.

According to an exemplary embodiment, the optical display 121 may be atransparent display or an opaque display. The transparent display is aninformation display apparatus in which a rear surface of a screendisplaying information is transparent. The transparent display includesa transparent device, and transparency may be adjusted by adjustinglight transmittance of the transparent device or adjusting an RGB valueof each pixel.

When the optical display 121 forms a touch screen by forming a layerstructure with a touch pad, the optical display 121 may be used as aninput device as well as an output device. The touch screen may detect atouch gesture of a user on the touch screen, and transmit informationabout the touch gesture to the controller 130. Examples of the touchgesture include tap, touch and hold, double tap, drag, panning, flick,drag and drop, and swipe.

The optical display 121 may include at least one of a liquid crystaldisplay, a thin-film transistor-liquid crystal display, an organiclight-emitting diode, a flexible display, a 3D display, and anelectrophoretic display. Also, the wearable device 100 may include atleast two of the optical displays 121 according to a structure of thewearable device 100.

The sound output 122 outputs audio data received from the communicator150 or stored in the memory 160. Also, the sound output 122 outputs asound signal related to a function performed by the wearable device 100,such as a call signal reception sound or a message reception sound. Thesound output 122 may include a speaker or a buzzer.

According to an exemplary embodiment, when an input is generated througha virtual input interface, the sound output 122 may output an audiosignal corresponding to the input.

The vibration motor 123 may output a vibration signal. For example, thevibration motor 123 may output a vibration signal corresponding to anoutput of audio data or video data, such as a call signal receptionsound or a message reception sound. Also, when an input is generatedthrough a virtual input interface, the vibration motor 123 may output avibration signal.

The controller 130 generally controls overall operations of the wearabledevice 100. For example, the controller 130 may execute programs storedin the memory 160 to control the sensor 110, the outputter 120, the userinput 140, the communicator 150, and the memory 160.

The controller 130 may set an input region based on a gesture recognizedby the image sensor 111. For example, when the image sensor 111recognizes a gesture of drawing a figure in the air or on an actualobject, the controller 130 may set a region corresponding to the figureas an input region.

The controller 130 may determine a virtual input interface to bedisplayed on the optical display 121 based on attributes of an inputregion.

The controller 130 may determine a type of a virtual input interfacebased on a first depth value of an input region, and display the virtualinput interface on the optical display 121 to overlap the input region.

The controller 130 may determine a type of a virtual input interfacebased on a type of an actual object where an input region is set, anddisplay the virtual input interface on the optical display 121 tooverlap the input region.

The controller 130 may determine a type of a virtual input interfacebased on a type of a gesture setting an input region or a size of theinput region, and display the virtual input interface on the opticaldisplay 121 to overlap the input region.

The controller 130 may determine a virtual input interface based on atype of an application being executed by the wearable device 100, anddisplay the virtual input interface on the optical display 121 tooverlap an input region.

The controller 130 may display a virtual input interface on atransparent display such that the virtual input interface is displayedon an input region observed through the transparent display.

The controller 130 may generate a second image in which a virtual inputinterface overlaps an input region included in a first image, anddisplay the second image including the virtual input interface on theoptical display 121.

The controller 130 may determine whether an input is generated through avirtual input interface based on a result of comparing a first depthvalue and a second depth value. For example, the controller 130 maydetermine that an input is generated through a virtual input interfacewhen a difference between first and second depth values is within athreshold value.

When a second depth value is greater than a first depth value thecontroller 130 may determine that an input is generated through avirtual input interface.

The controller 130 may control the outputter 120 to output anotification signal corresponding to generation of an input.

A user inputs data via the user input 140 to control the wearable device100. For example, the user input 140 may be a keypad, a dome switch, atouch pad (contact capacitance type, pressure resistance film type,infrared light detecting type, acoustic surface wave conducting type,integral tension measuring type, or a piezo-effect type), a jog wheel,or a jog switch, but is not limited thereto. According to an exemplaryembodiment, the user input 140 may include a virtual input interface.

The communicator 150 may include at least one component enabling thewearable device 100 to communicate with an external device or a server.For example, the communicator 150 may include a local area networker151, a mobile communicator 152, and a broadcast receiver 153.

The local area networker 151 may be a Bluetooth communicator, a nearfield communication/radio frequency identification (NFC/RFID) unit, awireless local area network (WiFi) communicator, a Zigbee communicator,an infrared data association (IrDA) communicator, an ultra wideband(UWB) communicator, or an Ant+ communicator, but is not limited thereto.

For example, the local area networker 151 may receive locationinformation of a second wearable device or a third wearable device.

The mobile communicator 152 transmits and receives a wireless signal toand from at least one of a base station, an external terminal, and aserver, on a mobile communication network. Here, the wireless signal mayinclude various types of data according to transmission and reception ofa voice call signal, an image call signal, or a text/multimedia message.

The broadcast receiver 153 receives a broadcast signal and/orinformation related to a broadcast from an external source through abroadcast channel. The broadcast channel may be a satellite channel or aterrestrial wave channel. According to an exemplary embodiment, thewearable device 100 may not include the broadcast receiver 153.

The memory 160 may store a program for processes and control of thecontroller 130, and may store input/output data, such as gestureinformation corresponding to an input mode, a virtual input interface,data input through a virtual input interface, sensing informationmeasured by a sensor, and content.

The memory 160 may include at least one of a flash memory, a hard disk,a micro type multimedia card, a card type memory, such as a securedigital (SD) or extreme digital (XD) memory, a random access memory(RAM), a static random access memory (SRAM), a read-only memory (ROM),an electrically erasable programmable read-only memory (EEPROM), aprogrammable read-only memory (PROM), a magnetic memory, a magneticdisk, and an optical disk. Also, the wearable device 100 may operate aweb storage or a cloud server that performs a storage function of thememory 160 on the Internet.

Programs stored in the memory 160 may be classified into a plurality ofmodules based on functions, such as a UI module 161 and a notificationmodule 162.

The UI module 161 may provide a specialized UI or GUI interworking withthe wearable device 100, according to applications. Also, according toan exemplary embodiment, the UI module 161 may select and provide avirtual input interface based on situations.

The notification module 162 may generate a signal for notifyinggeneration of an event in the wearable device 100. Examples of the eventgenerated in the wearable device 100 may include call signal reception,message reception, an input of a key signal through a virtual inputinterface, and schedule notification. The notification module 162 mayoutput a notification signal in the form of a video signal through theoptical display 121, in the form of an audio signal through the soundoutput 122, or in the form of a vibration signal through the vibrationmotor 123. Alternatively, the notification module 162 may output ahaptic signal by using an external wearable device, such as a ring, athimble, a bracelet, or a glove.

The methods described above may be recorded on a computer-readablerecording medium by being realized in computer programs to be executedby using various computers. The computer-readable recording medium mayinclude at least one of a program command, a data file, and a datastructure. The program commands recorded in the computer-readablerecording medium may be specially designed or well known to one ofordinary skill in the computer software field. Examples of thecomputer-readable recording medium include read-only memory (ROM),random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks,optical data storage devices, etc. Examples of the computer commandinclude mechanical codes prepared by a compiler, and high-levellanguages executable by a computer by using an interpreter.

As described above, according to one or more exemplary embodiments, thewearable device 100 may accurately determine whether an input isgenerated through a virtual input interface by comparing a depth valueof an input tool touching the virtual input interface, and a referencedepth value defined by a user.

While one or more exemplary embodiments have been described withreference to the figures, it will be understood by those of ordinaryskill in the art that various changes in form and details may be madetherein without departing from the spirit and scope as defined by thefollowing claims.

What is claimed is:
 1. A glass type wearable device comprising: an imagesensor; a display; and a processor configured to: control the imagesensor to capture one or more images to sense a gesture of a userselecting a partial region on a physical object by using an input tool;determine the partial region on the physical object as a user inputregion; determine a virtual input interface based on at least one of asize and a shape of the user input region; and control the display toprovide the determined virtual input interface based on the user inputregion, wherein the virtual input interface corresponds to the size andthe shape of the user input region.
 2. The glass type wearable device ofclaim 1, wherein the virtual input interface is determined based on atype of an application being executed by the glass type wearable device.3. The glass type wearable device of claim 1, wherein the displaycomprises a transparent display configured to provide the virtual inputinterface on a region of the transparent display corresponding to theuser input region as observed through the transparent display.
 4. Theglass type wearable device of claim 1, wherein the image sensor isconfigured to capture a first image of the user input region, and thedisplay is configured to display a second image of the virtual inputinterface over the user input region of the first image.
 5. The glasstype wearable device of claim 1, further comprising: a depth sensorconfigured to sense a first depth value corresponding to a distance fromthe glass type wearable device to the user input region, and a seconddepth value corresponding to a distance from the glass type wearabledevice to the input tool, wherein the processor is further configured todetermine whether an input is generated through the virtual inputinterface based on the first depth value and the second depth value. 6.The glass type wearable device of claim 5, wherein a displayed size ofthe virtual input interface is determined based on the first depthvalue.
 7. The glass type wearable device of claim 5, wherein theprocessor is configured to determine that an input is generated throughthe virtual input interface when a difference between the first andsecond depth values is less than a threshold value.
 8. The glass typewearable device of claim 5, wherein the processor is configured todetermine that an input is generated through the virtual input interfacewhen the second depth value is greater than the first depth value. 9.The glass type wearable device of claim 1, wherein the processor isfurther configured to: compare a distance from the glass type wearabledevice to the user input region with a threshold value, and control thedisplay, based on the distance being less than the threshold value, toprovide a first type virtual input interface on the display in thedetermined user input region, and control the display, based on thedistance being greater than or equal to the threshold value, to providea second type virtual input interface on the display in the determineduser input region.
 10. The glass type wearable device of claim 1,wherein the processor is further configured to determine a type of theinput tool and to control the display to provide the virtual inputinterface based on the user input region, a type of the physical object,and the type of the input tool.
 11. A method of providing, by a glasstype wearable device, a virtual input interface, the method comprising:capturing one or more images to sense a gesture of a user selecting apartial region on a physical object by using an input tool; determiningthe partial region on the physical object as a user input region;determining the virtual input interface based on at least one of a sizeand a shape of the user input region; and providing the determinedvirtual input interface based on the user input region, wherein thevirtual input interface corresponds to the size and the shape of theuser input region.
 12. The method of claim 11, wherein the virtual inputinterface is determined based on a type of an application being executedby the glass type wearable device.
 13. The method of claim 11, whereinthe providing of the virtual input interface comprises: capturing afirst image of the user input region by using an image sensor;generating a second image of the virtual input interface; and displayingthe second image over the user input region of the first image.
 14. Themethod of claim 11, further comprising: obtaining a first depth valuecorresponding to a distance from the glass type wearable device to theuser input region, and a second depth value corresponding to a distancefrom the glass type wearable device to the input tool; and determiningwhether an input is generated through the virtual input interface basedon the first depth value and the second depth value.
 15. The method ofclaim 14, wherein a displayed size of the virtual input interface isdetermined based on a size of the first depth value.
 16. The method ofclaim 14, wherein the determining of whether the input is generatedcomprises determining that a difference between the first and seconddepth values is less than a threshold value.
 17. The method of claim 14,wherein the determining of whether the input is generated comprisesdetermining that the second depth value is greater than the first depthvalue.
 18. The method of claim 11, further comprising: comparing adistance from the glass type wearable device to the user input regionwith a threshold value; based on the distance being less than thethreshold value, providing a first type virtual input interface in theuser input region; and based on the distance being greater than or equalto the threshold value, providing a second type virtual input interfacein the user input region.
 19. The method of claim 11, furthercomprising: determining a type of the input tool; and providing thevirtual input interface based on the user input region, a type of thephysical object, and the type of the input tool.
 20. A computer programproduct comprising a computer readable storage medium having a computerreadable program stored therein, wherein the computer readable program,when executed on a computing device of a glass type wearable device,causes the computing device to: capture one or more images to sense agesture of a user selecting a partial region on a physical object byusing an input tool; determine the partial region on the physical objectas a user input region; determine a virtual input interface based on atleast one of a size and a shape of the user input region; and providethe determined virtual input interface, wherein the virtual inputinterface corresponds to the size and the shape of the user inputregion.